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Formative  Evaluation  Plan: 

Computer-Based  Instructional  Modules  for  the  Forward  Support 

Battalion 


I.  purpose 


This  report  documents  the  formative  evaluation  process  to  be  applied  to  the  Computer- 
Based  Instructional  Modules  being  developed  to  instruct  personnel  in  the  Forward  Support 
Battalion  (FSB).  Subsequent  sections  of  this  report  introduce  formative  evaluation  and  related 
concepts,  describe  the  sequence  of  evaluation  steps  to  be  used  in  detail,  and  provide  a  schedule 
for  executing  these  steps. 


11.  FORMATIVE  EVALUATION 


All  CBI  courses  developed  for  die  FSB  are  to  be  subject  to  a  Formative  Evaluation. 
Bloom,  Hastings  and  Madaus  {Handbook  on  Formative  and  Summative  Evaluation  of  Student 
Learning,  New  York:  McGraw-Hill,  1971)  describe  formative  evaluation  of  a  curriculum  in  this 
way: 


.  .  .  formative  evaluation  involves  the  collection  of  appropriate 
evidence  during  the  construction  and  trying  out  of  a  new 
curriculum  in  such  a  way  that  revisions  of  the  curriculum  can  be 
based  on  die  evidence,  (p.  117)‘ 


Formative  evaluation  begins  in  the  design  phase  where  examination  of  the  proposed 
instructional  sequence  and  choice  of  materials  and  examples  can  be  reviewed  to  determine  if  they 
are  suitable.  In  the  early  developmental  phases  subject  matter  experts  and  instructional  designers 
can  examine  the  content  and  sequencing  to  determine  whether  the  instruction  seems  appropriate. 
As  instructional  sequences  are  completed,  they  can  be  tested  on  real  subjects  to  determine 
whether  they  proceed  through  the  material  in  the  anticipated  manner.  Finally,  the  subjects  can 
be  examined  to  determine  if  they  have  learned  from  the  prototype  materials.  The  results  of  this 
testing  can  be  used  to  diagnose  flaws  that  require  revision  to  the  instruction. 


^  This  process  is  analogous  to  quality  assurance  and  testing  practices  used  to  manufacture  products.  Starting 
with  the  design  phase,  the  suitability  of  the  form  and  materials  to  the  intended  function  is  examined  continuously. 
Wind  mnnel  tests  or  "brassboard"  tests  of  electronic  circuits  illustrate  some  of  the  early  tests  of  whether  the  concept 
or  principles  that  motivated  the  design  can,  in  fact,  be  realized  in  hardware.  Once  the  concept  is  demonstrated,  then 
the  process  of  fabrication  needs  to  be  examined  to  be  sure  that  it  does  not  introduce  unanticipated  flaws.  Similar 
principles  of  evaluation  can  be  applied  to  software  development  projects  and  have  led  to  the  development  of  software 
engineering  environments  in  which  the  interaction  of  'modules'  of  code  is  carefully  planned  and  tested.  Product 
recalls  and  software  'fixes'  illustrate  that  these  procedures  are  not  foolproof. 
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These  procedures  are  difficult  to  perform  on  standard  classroom-based  instructional 
materials  such  as  textbooks:  teachers  may  prefer  to  change  the  sequence  of  lessons  to  coordinate 
with  material  in  another  course;  testing  intact  groups  of  subjects  is  costly;  and  considerable 
information  about  the  use  of  the  material  must  be  gathered  in  order  to  use  results  as  an 
appropriate  diagnostic  of  the  quality  of  the  instructional  material.  Self-paced,  interactive 
materials  lend  themselves  more  readily  to  these  procedures:  the  sequence  of  the  material  is  'hard¬ 
wired  in';  the  prototype  curriculum  can  be  tested  on  individuals;  test  results  reflect  on 
instructional  quality  in  a  straightforward  way. 

Computer-based  instruction  (CBI)  has  a  further  advantage  over  odier  self-paced, 
interactive  instructional  media  because  it  is  easy  to  record  student  responses  to  diagnostic  quizzes 
the  student  takes  to  obtain  feedback  on  his/her  understanding.  These  quizzes  are  often  used  to 
guide  the  selection  of  instructional  sequences:  the  student  is  sent  forward  to  the  next  topic,  or 
routed  to  remedial  material  depending  upon  the  responses  to  the  diagnostic  questions.  The 
instructional  developer  can  compare  responses  to  practice/diagnostic  items  to  related  post-test 
responses  to  determine  whether  the  student  had  apparently  mastered  the  objective  during  the 
instruction.  If  students  have  difficulty  with  a  particular  instructional  sequence,  or  if  they  appear 
to  forget,  the  instruction  itself  probably  needs  to  be  changed. 

III.  THE  FORMATIVE  EVALUATION  PROCESS 

Our  comprehensive  plan  for  formative  evaluation  can  be  broken  into  three  distinct  phases: 

Internal  QC  --  SME  reviews  of  the  storyboards  prepared  by  the  instructional  developers 
to  be  sure  that  the  content  is  correct  and  the  logic  of  presentation  is  appropriate. 

Alpha  testing  --  testing  in-house  to  determine  if  the  courseware  tracks  with  the 
storyboards  and  to  identify  flaws  in  the  logic  of  the  presentation  of,  or 
navigation  through,  the  material.  This  is  accomplished  by  having  an  SME  and/or 
developer  from  another  team  attempt  to  go  through  the  lesson,  noting  errors, 
ambiguities  and  inconsistencies  in  logic,  navigation,  look-and-feel,  etc.  Problems 
are  documented  and  fixes  are  made.  At  the  completion  of  this  phase  the 
instructional  development  team  believes  the  courseware  will  accomplish  die 
intended  instruction. 

Beta  testing  —  testing  using  real  subjects.  This  testing  is  intended  to  be  the  first  practical 
run  through  of  the  courseware.  Subjects  are  drawn  from  the  pool  of  target 
users  of  the  courseware.  We  have  divided  this  phase  into  two  segments.  In  the 
first  segment  we  focus  on  identifying  flaws  in  the  logic  of  presentation  and  errors 
that  subjects  make  in  navigation  through  the  course.  In  the  second  segment  we 
focus  on  identifying  weak  elements  in  the  instruction  so  that  they  can  be  revised. 
At  the  completion  of  this  phase,  the  courseware  is  ready  to  be  used  in  the  field. 

These  procedures  are  explained  in  further  detail  in  the  following  paragraphs. 
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A.  Internal  QC 


In  the  CSS/FSB  project  internal  QC  begins  as  soon  as  the  instructional  developer  receives 
the  task  analysis  from  the  subject  matter  expert  (SME).  As  the  instructional  developer  develops 
the  storyboard  there  is  continual  interaction  with  the  SME  about  the  nature  of  the  content,  the 
choice  of  presentation  medium,  and  the  sequencing  of  sub-topics.  The  SME  continually  monitors 
the  storyboard  for  accuracy  of  content  and  appropriateness  of  sequence. 

This  monitoring  continues  through  the  phase  of  courseware  authoring.  As  the  developer 
completes  segments  (sub-topics)  the  SME  makes  judgements  about  the  adequacy  of  the  content 
and  method  of  the  presentation.  Corrections  and  adjustments  are  continuous  throughout  this 
cycle. 

B.  Alpha  Testing 

Once  the  prototype  courseware  for  a  lesson  is  completed,  a  different  SME/developer  diad 
goes  dirough  the  lesson,  comparing  it  to  the  storyboards.  All  of  the  branching  logic  is  exercised 
in  this  trial  to  see  if  there  are  errors  in  the  navigation  or  sequencing.  This  test  team  also  notes 
any  ambiguities  that  could  confuse  students,  and  changes  in  the  'look  and  feel'  of  the  system  diat 
might  interfere  with  navigation  through  die  course  or  learning  from  the  presentations.  All  of  the 
practice  and  diagnostic  quizzes  are  exercised  to  be  sure  that  th^  lead  correctly  to  the  next  logical 
sequence  of  material.  Fixes  are  made  to  the  problems  noted,  and  the  test  is  repeated.  At  the  end 
of  this  procedure  the  instructional  development  team  releases  the  courseware  for  beta  testing. 

C.  Beta  Testing 

Instructional  development  methodologies  (such  as  the  Joint  Services  Instructional  Systems 
Development  [ISD]  model)  recognize  that  well-written,  technically  accurate  instructional  materials 
might  appear  to  be  effective  teaching  tools,  but  it  is  difficult  to  predict  how  students  will  react 
to  the  material.  Only  the  target  trainees  can  provide  evidence  about  instructional  effectiveness; 
hence  the  importance  of  beta  testing  as  part  of  the  formative  evaluation  process. 

The  two  segments  of  beta  testing  focus  on  the  same  concerns,  but  to  different  degrees: 


•  Can  the  student  navigate  dirough  the  courseware  successfully? 

•  Does  the  student  learn  from  the  material? 

The  first  segment  focuses  mostly  on  the  first  concern.  Clearly,  if  students  have  difficulties 
navigating  through  the  course,  they  can  not  be  expected  to  learn  from  the  material.  In  addition 
to  strictly  navigational  concerns,  it  is  important  to  determine  if  the  content  presentation  is 
obviously  incorrect  or  ambiguous.  Problems  like  these  will  reduce  the  value  of  the  pre-test  to 
post-test  comparisons. 

During  this  segment  of  beta  testing,  the  trainees  are  told  that  they  may  ask  the  observing 
SME  questions  when  they  encoimter  difficulty  and  the  SMEs  can  intrude  by  asking  why  a  trainee 
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responded  as  he  or  she  did  to  a  particular  practice  item.  The  SME  can  provide  additional 
information  to  help  the  trainee  over  the  rough  spot;  and  the  SME  will,  during  the  lesson,  as  well 
as  after  completion  of  the  post-test,  gather  interview  data  useful  in  making  the  subsequent  lesson 
revisions. 

The  primary  purpose  of  this  segment  of  beta  testing  is  to  learn  as  much  as  possible  about 
the  interaction  between  each  trainee  and  the  lesson.  Some  of  Ais  kind  of  information  is  revealed, 
of  course,  in  Ae  data  regarding  responses  required  during  Ae  lesson  and  on  Ae  post-test. 
Allowing  for  staff  intrusion  and  queries  aims  Arectly  at  Ae  ongoing  thought  processes  of  the 
trainee  working  Arough  Ae  lesson.  The  intrusions  may  be  prompted  by  such  inAcators  as 
expression,  body  language,  utterances,  and,  of  course,  their  requests  for  assistance  and  Aeir 
responses  to  queries  by  the  staff. 

We  have  planned  for  three  subjects  to  go  Arough  this  testing  in  sequence,  wiA  fixes  made 
to  Ae  courseware  after  each  nm-Arough.  This  helps  us  to  be  sure  Aat  in  fixing  one  error  we 
have  not  introduced  problems  elsewhere. 

The  second  segment  of  beta  testing  focuses  mostly  on  Ae  learning  outcome.  The  goal 
is  to  have  subjects  take  Ae  lessons  wiAout  intervention  from  a  member  of  the  development  team. 
Completing  a  lesson  imder  Aese  conditions  would  indicate  Aat  Ae  courseware  is  ready  to  be  sent 
to  the  field.  Furthermore,  comparing  the  pre-test  to  the  post-test  results  for  this  subject  would 
be  indicative  of  the  learning  that  Ae  courseware  produces.  Specific  weaknesses  can  be  diagnosed 
from  Ae  post-test  and  these  can  be  compared  to  Ae  subject's  performance  in  Ae  practice  and 
diagnostic  quizzes  for  that  section  of  material.  All  of  Ais  evidence  can  be  used  to  determine 
what  correction  needs  to  be  taken  to  improve  the  courseware. 

Obviously,  if  a  flaw  in  navigation  or  critical  ambiguity  has  eluded  Ae  earlier  testing,  Aen 
the  person  monitoring  Ae  test  subject  will  have  to  intervene  to  allow  Ae  subject  to  proceed. 
Results  from  a  trial  with  this  kind  of  problem  will  not  be  as  useful  in  determining  Ae  utility  of 
the  courseware. 

In  Ais  segment  of  beta  testing  Ae  observing  and  note-taking  SMEs  are  limited  to 
intruding  only  when  a  trainee  becomes  completely  lost.  The  purpose  is  to  determine  whether  Ae 
lessons  are  in  fact  ready  to  go  to  the  field  where  they  must  function  independently  of  Ae 
developers,  and  that  Aey  will  induce  learning.  In  addition,  Ae  time  required  for  the  subject  to 
complete  Ae  course  (incluAng  pre-  and  post-testing)  will  be  useful  information  for  training 
managers  wishing  to  use  Aese  course  materials. 

Our  plan,  with  respect  to  Ae  Lanes-based  lessons  has  been  to  test  about  three  additional 
subjects  during  Ais  segment  of  beta  testing.  Given  Ae  limited  number  of  subjects,  inferences 
about  course  quality  will  entail  a  more  qualitative  than  statistically  rigorous  quantitative  analysis. 
For  Ae  FSB  Officer  courses,  we  will  have  to  rely  on  even  smaller  samples  of  subjects. 
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IV.  BETA  TESTING  PROCEDURES  IN  THE  FSB  PROJECT 


The  lessons  are  designed  to  allow  a  trainee  to  determine,  after  taking  the  pre-test,  vdiich 
topic  or  topics  he  or  she  needs  to  take,  and  then  enter  a  lesson  and  take  only  those  topics. 
During  validation  trials,  however,  all  test  trainees  will  complete  all  segments  of  the  lesson,  to 
provide  test  data  on  the  complete  lesson. 

Beta  test  trials  will  be  conducted  on  individual  trainees  one  at  a  time.  After  each  trainee 
completes  the  pre-test  --  lesson  --  post-test  cycle,  the  post-test  results  and  the  lesson  responses 
—  as  well  as  observations  made  during  that  cycle  and  during  a  final  interview  —  are  analyzed. 
Weak  spots  in  the  lesson  sequence  and  structure  are  identified  and  the  development  team 
specifies  changes  to  be  made  to  the  lesson  materials.  Depending  on  their  nature,  those  changes 
can  be  made  immediately,  before  the  next  test  trainee  takes  that  lesson,  or  held  until  all  trainees 
have  completed  the  beta  testing.  This  sequence  of  test-revise-test  is  continued  with  additional 
trainees  until  each  lesson  meets  the  performance  criterion  established  for  it. 

Each  trainee  is  observed  while  proceeding  through  the  lesson  and  during  the  pre-  and 
post-tests.  Frowns,  gestures  of  frustration,  paging  back-and-  forth— all  indicate  problems  within 
the  lesson.  During  the  first  segment  of  beta  testing  trainees  are  queried  immediately  about  the 
nature  of  the  problem,  and  are  assisted  as  they  proceed  through  a  lesson.  In  the  second  segment, 
there  should  be  very  little  evidence  of  this  type  of  problem,  and  no  interventions  will  be  allowed. 
Completion  times  for  the  lesson  and  scores  on  the  post-test  are  also  recorded,  for  use  in 
describing  both  the  effectiveness  of  the  lessons  and  the  time  factors  that  should  be  allowed  for 
personnel  taking  the  lessons. 

Trainees  will  be  interviewed  following  each  lesson.  Some  of  the  interview  questions  will 
deal  with  subject  matter  and  objectives;  some  will  ask  the  trainee  to  identify  problems  or 
difficulties  encountered  during  the  lesson;  some  will  ask  which  topics  or  portions  of  the  lesson 
seemed  smoother  or  more  enjoyable;  some  will  ask  for  subjective  comments,  suggestions,  and 
criticisms.  The  focus  of  the  interviews  will  be  modified  by  feedback  from  the  early  interviews. 

These  tests  will  be  closely  observed,  and  notes  will  be  taken  to  indicate  any  apparent 
difficulties  encoimtered  by  each  trainee.  These  notes  are  valuable  sources  of  information  to  the 
developers  developing  modifications  to  the  courseware. 

The  development  staff  will  dociiment  everything  that  happens  during  the  testing  of  the 
lessons.  This  will  include  pre-  and  post-test  performance,  responses  to  lesson  exercises  and 
questions,  comments  by  trainees,  the  nature  and  content  of  all  SME  interventions,  and  interview 
information.  The  subsequent  analyses  and  revision  specifications  will  be  described,  and  tied  to 
the  data  listed  above,  for  each  trainee  trial.  In  this  way  a  documentation  track  will  show  how 
each  lesson  proceeded  through  the  validation  sequence.  The  guidelines  to  be  followed  by  project 
developers  and  SMEs  further  define  the  validation  procedures.  These  guidelines  appear  in  the 
Appendices  to  this  report. 
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V.  SCHEDULE  FOR  BETA  TESTING 


Given  adequate  availability  of  soldiers  who  are  members  of  the  target  audience  for  each 
lesson,  the  complete  series  of  beta  tests  and  revisions  for  a  lesson  can  be  completed  in  two 
calendar  weeks.  The  National  Guard  project  SMEs  will  obtain  test  soldiers  for  the  validation 
trials.  It  is  possible  that  the  number  of  test  trainees  might,  in  some  cases,  have  to  be  increased, 
if  necessary  to  evaluate  the  adequacy  of  revisions  made  during  validations  trials. 

The  following  schedule  is  based  on  the  delivery  schedule  in  the  proposed  modification 
to  the  delivery  order.  Some  lessons,  indicated  by  asterisks  in  the  schedule,  have  already  been 
through  preliminary  beta  testing.  Most  lessons  will  enter  the  test  sequence  before  the  'not  later 
than'  (NLT)  dates  shown.  The  schedule  identifies  the  enlisted  MOSs  and  the  officer  branch 
designation  representing  the  target  audiences  for  which  the  lessons  were  designed,  and  which  are 
required  for  validating  the  related  lessons. 

A.  Lessons  to  Begin  Beta  Tests  NLT  1  September 


Company/Lane 

Lesson 

Soldier  MOSs  Required 

Supply  Company 

Inspection  of  POL  products 

MOS  77F 

Maintenance  Company 

*  Inspect/Troubleshoot 

Tracked  Vehicle  Auto.  Sys. 

MOS  52,  63B/S/H 

Repair  Diesel  Power  Pack 

MOS  63H 

Medical  Company 

*  Plan  for  Evacuation 

MOS  67B,  91 B 

Officer  50A 

*  Control  Bleeding 

MOS  91 B 

Defend  Company  Sector 

Terrain  Analysis 

Co.  Grade  Officers, 

Sr.  NCOS 

*  Plan  Sector  Defense 

Co.  Grade  Officers, 

Sr.  NCOS 

B.  Lessons  to  Begin  Beta  Tests  NLT  1  October 


Company/Lane 

Lesson 

Soldier  MOSs  Required 

Supply  Company 

Tanker  Opns  and  Safety 

MOS  77F 

Maintenance  Company 

Repair  Traversing  Systems 

MOS  45K 

Organize/Dispatch  Maint. 

Support  Team 

MOS  52,  63 

Officers 

Medical  Company 

*  Triage/Survey  Patients 

MOS  91B/C 

Officer  67B,  60A 

Defend  Company  Sector 

Prepare  Support  Plan 

Co.  Grade  Officers, 

Sr.  NCOS 

Defend/Prepare  Level  1,  II,  III 
Threat  Attack 

Co.  Grade  Officers, 

Sr.  NCOS 

C.  Lessons  to  Begin  Beta  Tests  NLT  1  November 


Company/Lane 

Lesson 

Soldier  MOSs  Required 

Supply  Company 

Direct  POL  Environment/ 

Security  Controls 

MOS  77F 

Maintenance  Company 

Supervise/Perform  BDAR 

MOS  63H,  Officer 

Medical  Company 

General  Medical  Treatment 

MOS  91 B,  Officer  67B 

Treating  Respiratory 

Dysfunction 

MOS  91 B,  Officer  67B 

Defend  Company  Sector 

Perform  Withdrawal  Under  Fire 

Co.  Grade  Officers, 

Sr.  NCOS 

D.  Lessons  to  Begin  Beta  Tests  NLT  1  December 


Company/Lane 

Lesson 

Soldier  MOSs  Required 

Supply  Company 

Supervise  Receipt/Storage  POL 
Products 

MOS  77F 

Maintenance  Company 

Troubleshoot/Repair  Radio  Sets 

MOS  63H,  29E 

Repair  BFV  TOW  ISU 

MOS  63H,  27E 

Medical  Co. 

Casualty  Management 

MOS  91 B,  Officer  67B 

Defend  Company  Sector 

Conduct  Hasty  Displacement 

Co.  Grade  Officers, 

Sr.  NCOS 

Collective  Training  Lane 

Co.  Grade  Officers, 

Sr.  NCOS 

£.  Beta  Tests  of  Staff  Training: 

Appropriate  subjects  for  validation  trials  will  be  officers  and  NCOs  per  the  current  TO&E 
for  the  Forward  Support  Battalion  and  Support  Battalion.  These  tests  will  include  both  the  CBI 
and  computer-managed  instruction  (CMI)  lessons.  The  following  schedule  is  based  upon  receipt 
of  beta-tested  CBI  and  CMI  from  the  Battle  Staff  Training  System  Development  Project  for  all 
duty  positions  except  SPO  and  BMMO.  The  dates  shown  are  based  upon  receipt  of  these 
materials  at  least  two  weeks  in  advance  of  the  projected  beta  testing  start  date. 

Lessons  to  Begin  Beta  Tests  NLT  1  November;  Common  Core,  S2,  SPO 
Lessons  to  Begin  Beta  Tests  NLT  15  November:  SI,  S3 
Lessons  to  Begin  Beta  Tests  NLT  1  December:  S4,  XO 
Lessons  to  Begin  Beta  Tests  NLT  15  December:  S2/S3,  BMMO 
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VI.  APPENDIX  A 


BETA  TESTING  --  Segment  One 

1 .  Read  the  following  to  the  soldier: 

You  are  going  to  go  through  a  lesson  presented  by  the  computer.  First  I'll  describe  the 
lessons,  how  they  work,  and  what  you  are  to  do  as  you  proceed  through  them,  then  I'll  show  you 
how  to  use  die  computer  and  certain  aspects  of  the  software  programs.  Don't  worry,  if  you  aren't 
experienced  with  computers.  I'll  tell  you  about  the  computer  only  to  the  extent  that  you  need  to 
know  in  order  to  proceed. 

Each  lesson  is  designed  to  function  as  a  tutor  —  an  individual  source  of  instruction  just 
for  you.  Each  lesson  is  interactive  --  which  means  that  you  will  be  asked  to  use  information  as 
it  is  presented  to  you.  The  computer  does  not  simply  present  a  lecture:  it  will  ask  questions,  and 
you  will  enter  your  answer  using  the  computer  keyboard.  Some  questions  will  be 
multiple-choice,  some  matching,  and  sometimes  you  will  be  asked  to  enter  one  or  more  words, 
typing  them  in.  Each  time  you  make  an  entry,  the  computer  will  tell  you  whether  your  answer 
was  correct  or  incorrect.  Then  you  will  go  on  to  the  next  step.  This  interaction  between  you  and 
the  computer  lesson  will  help  you  learn. 

Before  you  begin  the  lesson,  you  will  take  a  test  designed  to  determine  how  much  you 
already  know  and  can  do.  The  computer  will  examine  your  test  results  and  indicate  which 
portions  of  the  lesson  you  can  skip  and  which  portions  you  should  take.  You  will  always  have 
the  choice  of  skipping  or  taking  a  lesson  segment.  You  might  want  to  review  something  you 
already  know,  because  that's  worthwhile  practice.  After  you  have  completed  the  lesson,  you  will 
take  a  post-test,  to  determine  what  you  have  learned,  and  what  might  require  going  through  the 
post  test  lesson  a  second  time.  Again,  the  computer  will  report  your  test  scores  to  you,  and 
record  the  results  in  our  record-keeping  system. 

You  are  helping  us  test  these  lessons.  We  want  you  to  help  us  find  the  "bugs"  in  them. 
We  want  you  to  tell  us  when  something  is  not  clear,  or  is  confusing,  or  when  you  have  problems 
of  any  kind.  One  of  us  will  be  watching  you  to  see  how  smoothly  the  lesson  works,  and  to 
identify  any  errors  you  make,  so  we  can  revise  the  lesson  so  the  next  soldier  won't  make  that 
error.  If  you  run  into  a  problem,  tell  one  of  us,  ask  us  for  clarification,  tell  us  what  isn't  clear. 
Chances  are  that  if  you  have  trouble,  it  isn't  your  fault,  the  fault  is  in  the  lesson.  Sometimes  we 
might  ask  you  a  question  if  we  diink  there  is  something  you  can  tell  us.  Try  to  relax.  The  lesson 
is  on  the  spot;  you  aren't. 

2.  Have  the  soldier  sit  before  the  computer.  Explain  the  use  of  the  mouse  and  any  other 

features  of  the  computer,  software,  and  any  imique  features  of  the  lesson. 

3.  Administer  the  pre-test.  Discuss  the  results  with  the  soldier.  Make  notes  on  any 

questions  the  soldier  asks.  We  will  use  this  information  to  prepare  a  brief  intro  for  future 

test  soldiers. 
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4.  Show  the  soldier  how  to  access  the  opening  frame  and  then  let  him  or  her  proceed. 


5.  Sit  where  you  can  see  the  soldier's  profile  and  the  monitor  screen.  Look  for  the 
following: 

a.  Frowns,  puzzled  looks,  head  shaking,  any  body  language  that  indicates 
some  form  of  difficulty,  taking  a  long  time  to  make  a  response.  When  you 
spot  one  of  these  actions,  ask  the  soldier  what's  wrong.  Note  everything 
he  or  she  says.  If  you  have  to  provide  additional  information  to  help  the 
soldier  continue,  note  the  information. 

b.  Errors  by  the  soldier.  Note  the  question  and  the  wrong  response. 

c.  If  a  soldier  asks  you  a  question,  note  the  question,  as  well  as  the  nature 
of  the  information  you  had  to  provide  to  get  past  the 
problem. 

d.  If  the  soldier  decides  to  skip  any  component  of  the  lesson,  check  the 
pre-test  scores  to  see  if  diat  option  was  offered  based  on  die  score. 

e.  Note  any  other  problems,  deviations  in  procedures,  and  all  comments  made 
by  the  soldier  about  any  part  of  the  lesson. 

6.  When  the  soldier  has  completed  the  lesson,  conduct  a  brief  interview,  asking  the 
following  questions: 

a.  How  did  you  like  the  lesson? 

b.  Did  the  lesson  feel  like  a  tutor  individually  trying  to  teach  you  the  subject 
matter? 

c.  Were  the  directions  always  clear?  Could  you  always  tell  exactly  what  you 
were  being  asked  to  do? 

d.  Was  the  level  of  the  vocabulary  about  right?  Or  did  the  lesson  use  words 

that  made  the  lesson  difficult? 

e.  Did  the  lesson  use  any  names  or  words  or  technical  terms  that  were 
confusing?  If  so,  can  you  give  an  example? 

f  Was  the  feedback  helpful—the  information  the  lesson  gave  you  after  you 
answered  a  question  or  did  a  task?  How  might  it  have  been  more  helpful? 

g.  Do  you  have  any  suggestions  on  how  we  can  make  the  lesson  more 
interesting,  easier  to  understand,  or  how  we  can  make  it  more  effective? 

7.  Administer  the  post-test.  Cjo  over  the  results  with  the  soldier.  As  you  do,  look  for 
any  additional  information  from  the  soldier  that  can  help  us  improve  the  lesson. 

8.  Rewrite  your  notes  if  necessary  to  make  them  easily  readable.  Add  any  additional 
clarification  or  ideas  that  occur  to  you. 

9.  Complete  the  attached  form. 
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VII.  APPENDIX  B 


BETA  TESTING  --  Segment  Two 

1 .  Read  the  following  to  the  soldier: 

You  are  going  to  go  through  a  lesson  presented  by  the  computer.  First  I'll  describe  the 
lessons,  how  they  work,  and  what  you  are  to  do  as  you  proceed  through  them,  then  I'll  show  you 
how  to  use  the  computer  and  certain  aspects  of  the  software  programs.  Don't  worry,  if  you  aren't 
experienced  with  computers.  I'll  tell  you  about  the  computer  only  to  the  extent  diat  you  need  to 
know  in  order  to  proceed. 

Each  lesson  is  designed  to  function  as  a  tutor  --  an  individual  source  of  instruction  just 
for  you.  Each  lesson  is  interactive  —  which  means  that  you  will  be  asked  to  use  information  as 
it  is  presented  to  you.  The  computer  does  not  simply  present  a  lecture:  it  will  ask  questions,  and 
you  will  enter  your  answer  using  the  computer  keyboard.  Some  questions  will  be 
multiple-choice,  some  matching,  and  sometimes  you  will  be  asked  to  enter  one  or  more  words, 
typing  them  in.  Each  time  you  make  an  entry,  the  computer  will  tell  you  whether  your  answer 
was  correct  or  incorrect.  Then  you  will  go  on  to  the  next  step.  This  interaction  between  you  and 
the  computer  lesson  will  help  you  learn. 

Before  you  begin  the  lesson,  you  will  take  a  test  designed  to  determine  how  much  you 
already  know  and  can  do.  The  computer  will  examine  your  test  results  and  indicate  which 
portions  of  the  lesson  you  can  skip  and  which  portions  you  should  take.  You  will  always  have 
the  choice  of  skipping  or  taking  a  lesson  segment.  You  might  want  to  review  something  you 
already  know,  because  that's  worthwhile  practice.  After  you  have  completed  the  lesson,  you  will 
take  a  post-test,  to  determine  what  you  have  learned,  and  what  might  require  going  through  the 
post  test  lesson  a  second  time.  Again,  the  computer  will  report  your  test  scores  to  you,  and 
record  the  results  in  our  record-keeping  system. 

You  are  helping  us  test  these  lessons.  We  want  you  to  help  us  find  the  "bugs"  in  them. 
We  want  you  to  tell  us  when  something  is  not  clear,  or  is  confusing,  or  when  you  have  problems 
of  any  kind.  One  of  us  will  be  watching  you  to  see  how  smoothly  the  lesson  works,  and  to 
identify  any  errors  you  make,  so  we  can  revise  the  lesson  so  the  next  soldier  won't  make  that 
error.  If  you  run  into  a  problem,  work  your  way  through  it  if  you  can,  then  we'll  discuss  it  later 
“  don't  ask  us  any  questions  until  later.  Chances  are  that  if  you  have  trouble,  it  isn't  your  fault, 
the  fault  is  in  the  lesson.  Try  to  relax.  The  lesson  is  on  the  spot;  you  aren't. 

2.  Have  the  soldier  sit  before  the  computer.  Explain  the  use  of  the  mouse  and  any  other 

features  of  the  computer,  software,  and  any  imique  features  of  the  lesson. 

3.  Administer  the  pre-test.  Discuss  the  results  with  the  soldier.  Make  notes  on  any 

questions  the  soldier  asks.  We  will  use  this  information  to  prepare  a  brief  intro  for  future 

test  soldiers. 

4.  Show  the  soldier  how  to  access  the  opening  frame  and  then  let  him  or  her  proceed. 
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5. Sit  where  you  can  see  the  soldier's  profile  and  the  monitor  screen.  Look  for  the 
following: 

a.  Frowns,  puzzled  looks,  head  shaking,  any  body  language  that  indicates 
some  form  of  difficulty,  taking  a  long  time  to  make  a  response.  When  you 
spot  one  of  these  actions,  ask  the  soldier  what's  wrong.  Note  everything 
he  or  she  says.  If  you  have  to  provide  additional  information  to  help  the 
soldier  continue,  note  the  information. 

b.  Errors  by  the  soldier.  Note  the  question  and  the  wrong  response. 

c.  If  a  soldier  asks  you  a  question,  note  the  question,  as  well  as  the  nature  of 
the  information  you  had  to  provide  to  get  past  the  problem. 

d.  If  the  soldier  decides  to  skip  any  component  of  the  lesson,  check  the 
pre-test  scores  to  see  if  diat  option  was  offered  based  on  the  score. 

e.  Note  any  other  problems,  deviations  in  procedmes,  and  all  comments  made 
by  the  soldier  about  any  part  of  the  lesson. 

6.  When  the  soldier  has  completed  the  lesson,  conduct  a  brief  interview,  asking  the 
following  questions: 

a.  How  did  you  like  the  lesson? 

b.  Did  the  lesson  feel  like  a  tutor  individually  trying  to  teach  you  the  subject 
matter? 

c.  Were  the  directions  always  clear?  Could  you  always  tell  exactly  what  you 
were  being  asked  to  do? 

d.  Was  the  level  of  the  vocabulary  about  right?  Or  did  the  lesson  use  words 
that  made  the  lesson  difficult? 

e.  Did  the  lesson  use  any  names  or  words  or  technical  terms  that  were 
confusing?  If  so,  can  you  give  an  example? 

f  Was  the  feedback  helpful— the  information  the  lesson  gave  you  after  you 
answered  a  question  or  did  a  task?  How  might  it  have  been  more  helpful? 

g.  Do  you  have  any  suggestions  on  how  we  can  make  the  lesson  more 
interesting,  easier  to  understand,  or  how  we  can  make  it  more  effective? 

7.  Administer  the  post-test.  Go  over  the  results  with  the  soldier.  As  you  do,  look  for  any 
additional  information  from  the  soldier  that  can  help  us  improve  the  lesson. 

8.  Rewrite  your  notes  if  necessary  to  make  them  easily  readable.  Add  any  additional 
clarification  or  ideas  that  occur  to  you. 

9. Complete  the  attached  form. 
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VIII.  CBI  LESSON  BETA  TESTS 


_ BETA  1  _ BETA  2 

TITLE  OF  LESSON  _ _ 

NAME  OF  PERSON  CONDUCTING  TEST _ ^DATE _ 

SOLDIER'S  NAME  _  RANK  _  MOS 

POSITION  _  UNIT/LOCATION  _ 

PRE-TEST  SCORE  _  #  ITEMS  MISSED:  _ 

POST-TEST  SCORE  _  #  ITEMS  MISSED:  _ _ 

- Timing  Information - 

Pre-test  start  time  _  finish  time  _  Elapsed  time  (minutes) 

Lesson  start  time  _  finish  time  _  Elapsed  time  (minutes) 

Post-test  start  time  _  finish  time  _  Elapsed  time  (minutes) 


COMMENTS  ON  POST-TEST  ITEMS  MISSED  (Interviewer  will  determine,  as  far  as 
possible,  why  the  soldier  answered  incorrectly,  or  did  not  know  the  correct 
answer.  Use  the  NOTE  section  to  record  this  information.) 
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IX.  QUESTIONNAIRE  (After  post-test) 

1.  How  did  you  like  the  lesson? 


2.  Did  the  lesson  feel  like  a  tutor  individually  trying  to  teach 
you  the  subject  matter? 


3.  Were  the  directions  always  clear?  Could  you  always  tell  exactly 
what  you  were  being  asked  to  do? 


4.  Was  the  level  of  the  vocabulary  about  right? 


5.  Did  the  lesson  use  any  names  or  technical  terms  that  were 
confusing?  If  so,  what  were  they? 


6.  Was  the  feedback  helpful  the  information  the  lesson  gave  you 
after  you  answered  a  question  or  did  a  task?  How  might  it  have 
been  more  helpful? 


7.  Do  you  have  any  suggestions  on  how  we  can  make  the  lesson  more 
interesting,  easier  to  understand,  or  how  we  can  make  it  more 
effective? 
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